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Abstract 


To  improve  operational  effectiveness  for  the  Canadian  Forces  (CF),  the  Joint  Unmanned  Aerial 
Vehicle  Surveillance  Target  Acquisition  System  (JUSTAS)  project  is  acquiring  a  medium 
altitude,  long-endurance  (MALE)  uninhabited  aerial  vehicle  (UAV).  In  support  of  the  JUSTAS 
project,  Defence  Research  and  Development  Canada  (DRDC)  -  Toronto  is  investigating  the 
human  factors  issues  of  UAV  ground  control  stations  (GCS)  interfaces  for  UAVs  and  exploring 
possible  solutions  using  multimodal  displays.  This  progress  report  provides  an  update  project 
from  March  of  201 1  to  December  of  201 1  of  this  Baseline  and  Multimodal  UAV  GCS  Interface 
Design.  The  report  discusses  the  pilot  testing  of  the  baseline  condition  of  the  experiment.  In 
addition,  the  progress  of  the  multimodal  display  design  is  provided.  The  implementation  of  the 
multimodal  display  design  to  the  experiment  is  also  addressed. 


n 


Executive  summary 


Baseline  and  Multimodal  UAV  GCS  Interface  Design:  Progress 
Report 

[Wayne  Giang,  Yeti  Li,  Catherine  Burns];  DRDC  Toronto  TR  [enter  number 
only:  9999-999];  July  2013. 

Background:  Uninhabited  aerial  vehicles  (UAVs)  are  remotely  controlled  aircraft  used  for  a 
variety  of  civilian  and  military  applications  including  command,  control,  communications, 
computers,  intelligence,  surveillance  and  reconnaissance  (C4ISR).  To  improve  C4ISR  capability, 
the  Canadian  Forces  (CF)  is  acquiring  a  medium-altitude,  long-endurance  (MALE)  UAV  under 
the  Joint  Unmanned  Aerial  Vehicle  Surveillance  Target  Acquisition  System  (JUSTAS)  project.  In 
support  of  the  JUSTAS  project,  Defence  Research  and  Development  Canada  (DRDC)  -  Toronto 
is  investigating  the  human  factors  issues  of  UAV  ground  control  stations  (GCS)  interfaces  for 
UAVs  and  exploring  possible  solutions  using  multimodal  displays.  This  project,  the  Baseline  and 
Multimodal  UAV  GCS  Interface  Design,  is  ongoing  as  of  December  of  201 1. 

Results:  As  of  December  of  201 1,  the  pilot  testing  for  the  baseline  condition  of  the  study  was 
completed.  Several  issues  of  the  experimental  environment  were  identified  and  resolved.  The 
baseline  condition  of  the  GCS  simulator  was  stable  and  ready  to  run  participants.  Automated 
methods  for  aggregating  and  analyzing  the  data  were  being  created  to  accelerate  the  process  of 
data  collection  and  preliminary  data  analysis. 

The  multimodal  display  for  the  experiment  was  being  developed.  The  report  described  the 
finalized  auditory  and  tactile  displays  and  their  implementation  into  the  multimodal  condition  of 
the  experiment.  The  auditory  sonification  of  engine  RPM  and  auditory  warnings  were  designed 
and  introduced  to  the  experiment.  The  tactile  display  to  present  attitude  upset  of  the  UAV  was 
being  developed.  This  additional  experiment  was  planned  and  designed  to  compare  the  sequential 
and  equalizer  attitude  upset  presentation.  The  effectiveness  of  various  tactile  displays  will  be 
evaluated  to  examine  how  well  they  are  able  to  communicate  levels  of  tactile  upset. 

Significance:  The  report  discusses  the  completion  of  the  pilot  testing  for  the  baseline  condition  of 
the  experiment.  The  significance  of  the  multimodal  display  design  and  the  implementation  of  the 
multimodal  display  design  to  the  experiment  are  also  addressed. 

Future  plans:  In  early  2012,  the  future  plans  of  the  project  were  to  complete  running  of  the 
baseline  GCS  condition  for  participants  and  finalize  the  designs  of  the  multimodal  displays, 
including  running  of  the  tactile  experiment. 
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1  Introduction 


An  uninhabited  aerial  vehicle  (UAV)  is  an  aircraft  system  without  an  onboard  pilot  or  crew.  The 
UAV  is  controlled  from  a  ground  control  station  (GCS).  Today's  UAVs  are  highly  automated  and 
to  some  extent,  autonomous.  UAVs  can  be  directed  to  follow  a  pre-programmed  mission;  they 
can  fly  to  designated  waypoints,  fly  specific  patterns,  correct  for  course  deviations  and  hold 
above  a  particular  coordinate  or  target.  Some  UAVs  can  perform  automated  take-off  and  landing 
(e.g.,  the  CU-170  Heron  used  by  the  Canadian  Forces  during  the  war  in  Afghanistan).  UAV 
developers  argue  that  automation  and  autonomy  provide  several  benefits:  (a)  increased  flight 
safety;  (b)  simplified  operations;  (c)  lower  operating  costs;  and  (d)  reduced  operator  workload 
(Attar,  2005).  However,  these  benefits  are  not  always  realized.  Along  with  the  benefits  of 
automation,  some  disadvantages  occur  such  as  loss  of  situation  awareness  (Endsley  and  Kiris, 
1995;  Endsley,  1996),  loss  of  supervisory  control  (Parasuraman,  Molloy,  Mouloua,  &  Hilbum, 
1996;  Sheridan,  1987),  information  deprivation  that  occurs  from  remote  operations  (Manning, 
Rash,  LeDuc,  Noback,  &  McKeon,  2004),  and  high  workload  levels  for  operators  (Lee,  2008; 
Woods,  1996).  These  issues  point  to  the  need  for  improved  interfaces  to  help  these  operators 
remain  in  the  loop  and  maintain  situation  awareness  during  the  remote  monitoring  tasks  typical  of 
UAV  monitoring. 

The  work  described  in  this  report  is  in  support  of  the  Canadian  Forces  (CF)  Joint  Unmanned 
Aerial  Vehicle  Surveillance  Target  Acquisition  System  (JUSTAS)  project.  The  JUSTAS  project 
entails  the  acquisition  of  a  long  endurance  UAV.  This  work  is  directed  towards  understanding 
the  human  monitoring  challenges  with  UAVs  similar  to  the  UAVs  that  could  be  acquired  through 
the  JUSTAS  project.  Further,  this  project  will  explore  the  use  of  multimodal  interfaces  for  UAV 
control  leading  to  new  design  criteria  for  UAV  ground  control  stations,  or  improved  requirements 
for  future  acquisitions. 

This  work  builds  on  the  work  of  Giang  et  al.,  2010  which  reviewed  the  current  research  on 
multimodal  displays  and  ecological  displays  for  UAV  control.  This  report  is  a  mid-project 
progress  report  detailing  the  work  that  has  occurred  in  order  to  prepare  for  a  large  scale  study  of 
two  GCS  designs.  One  of  the  GCS  designs  will  be  a  visual  interface  (called  the  “baseline 
condition”),  essentially  simulating  a  typical  UAV  visual  interface.  The  other  design  will  be  a 
multimodal  interface  where  tactile  and  auditory  information  will  be  added  to  the  interface  to  see  if 
this  new  information  can  improve  operator  performance  and  situation  awareness. 

A  number  of  goals  have  been  accomplished  over  the  last  few  months,  since  the  submission  of  our 
last  contract  report  in  March  2011,  to  support  the  design  and  testing  of  a  multimodal  UAV  GCS 
simulator.  This  report  will  start  with  a  brief  review  of  the  past  achievements  in  our  last  contract 
report  in  March,  2011.  This  report  will  also  outline  the  progress  in  two  activities:  the  pilot  testing 
of  the  baseline  condition  of  the  experiment  and  the  design  of  the  multimodal  GCS  interface.  The 
goal  of  this  report  is  to  provide  a  short  summary  of  the  results  achieved  in  these  two  activities. 


2  Pilot  testing  of  the  Baseline  condition  of  the 
Experiment 


In  April  of  20 1 1 ,  pilot  testing  of  the  baseline  condition  of  the  experiment  was  started.  This  pilot 
testing  was  performed  to  examine  the  suitability  of  the  experimental  guidelines  and  to  evaluate 
the  stability  of  the  experimental  platform.  The  experiment  was  conducted  with  two  participant 
groups.  The  naive  participant  group  had  no  pilot  experience  and  the  expert  participant  group  had 
recently  acquired  at  least  ten  flying  hours.  Over  the  last  8  months,  10  naive  participants  in  the 
GCS  baseline  condition  were  successfully  tested.  However,  a  number  of  complications  occurred 
during  this  pilot  testing  that  delayed  the  running  of  the  experiment  until  the  beginning  of  2012.  In 
late  October  2011,  the  experiment  room  was  closed  for  renovation.  The  experiment  equipment 
was  moved  to  an  alternate  room.  This  resulted  in  a  small  delay  in  running  the  baseline  condition. 
In  the  middle  of  November  201 1,  we  encountered  a  technical  problem  on  the  eye  tracking  system, 
and  the  troubleshooting  and  verification  lasted  until  late  December  2011.  The  pilot  test  showed 
that  overall  our  experimental  guidelines  and  procedures  were  effective.  However,  a  number  of 
modifications  to  the  procedure  were  necessary.  Our  situational  awareness  questionnaire  which 
was  implemented  during  the  experimental  session  had  a  number  of  questions  that  were  confusing 
to  our  pilot  participants.  The  situational  awareness  questions  were  clarified  based  on  suggestions 
from  the  pilot  participants.  Additionally,  a  practice  scenario  in  which  the  UAV  lands  safely  was 
added  to  the  training  phase.  Therefore,  two  practice  scenarios  are  now  provided  to  the  participants 
during  the  training  phase,  one  where  the  participant  is  required  to  abort  the  landing  and  one  where 
the  UAV  can  land  safely.  The  additional  practice  scenario  was  added  to  remove  the  bias  of 
experiencing  only  an  abort  scenario  during  the  training  phase.  A  new  visual  indicator  of  engine 
revolutions  per  minute  (RPM)  was  added  on  the  visual  interface  of  the  UAV  landing  display,  and 
the  participant  was  required  to  monitor  this  indicator  during  the  scenario.  An  additional  sheet  of 
instructions  was  also  provided  after  the  training  video  to  provide  participants  with  information 
regarding  the  RPM  indicator 

Aside  from  stability  of  the  GCS  experimental  platform,  there  were  two  other  challenges 
encountered  during  the  pilot  study.  One  challenge  was  that  participants  with  glasses  could  not  be 
reliably  calibrated  using  the  eye  tracking  system.  The  experimental  protocol  was  changed  to 
recruit  only  those  who  had  normal  vision  or  who  had  corrected  to  normal  vision  using  contact 
lenses.  Another  challenge  was  in  scheduling  of  participants.  Each  participant  was  required  to 
come  in  for  three  2  hour  sessions  in  the  course  of  one  working  week.  This  made  scheduling 
difficult  with  three  time-slots  per  day,  and  only  15  possible  slots  each  week.  The  experimental 
procedure  was  changed  to  allow  for  three  sessions  within  7  days  instead  of  one  working  week  to 
make  scheduling  easier,  but  schedule  changes  and  cancellations  late  in  the  week  can  still  resulted 
in  scheduling  conflicts. 

During  the  pilot  test,  several  problems  with  the  experimental  platform  were  also  encountered  that 
led  to  the  loss  of  data  during  some  of  the  pilot  sessions.  After  running  the  10  participants,  almost 
30%  of  the  data  collected  was  affected  by  some  aspect  of  the  experimental  platform  crashing;  in 
fact,  each  of  our  1 0  participants  had  at  least  one  scenario  which  resulted  in  some  loss  of  data.  The 
errors  encountered  involved  a  number  of  components  in  the  GCS  simulator  including  the 
experimenter  console,  the  x-plane  simulation,  the  secondary  task  and  situation  awareness 
questions,  and  the  eye  tracking  system.  In  the  subsequent  months,  the  team  debugged  and  fixed 
many  of  the  errors.  As  of  December,  2011,  all  of  the  problems  that  were  encountered  during  the 
pilot  testing  were  resolved  and  testing  of  the  GCS  baseline  condition  was  ready  to  begin. 


The  results  of  the  pilot  test  indicated  that  participants  were  able  to  respond  to  most  of  the 
simulation  events.  Participants  were  instructed  to  press  the  button  labelled  “Operator  Concern”  if 
they  detected  any  critical  events  (e.g.,  wind  shear,  turbulence,  or  RPM  warnings).  For  the  10  pilot 
subjects,  300  events  occurred.  Out  of  300  of  such  events,  only  2  (0.667%)  events  did  not  receive 
any  responses  and  19  (6.33%)  events  had  responses  that  occurred  longer  than  10  seconds  from  the 
onset  of  the  event.  The  average  response  time  for  the  participants  pressing  the  operator  concern 
button  after  the  start  of  an  event  was  4.5  seconds.  In  the  secondary  task,  a  trial  was  presented 
every  2  seconds.  The  participant  was  required  to  respond  during  this  window.  A  successful 
response  on  the  true  target  was  regarded  as  a  hit,  while  a  response  on  a  false  target  was  judged  as 
a  false  alarm.  In  this  simulation  of  a  wind  shear  monitoring  task,  participants  had  an  average  hit 
percentage  of  78.84%  and  a  false  alarm  percentage  of  7. 12%. 

Most  of  this  preliminary  data  analysis  was  done  manually  by  calculating  and  extracting  values 
from  the  experiment  log  files,  which  was  a  very  time  consuming  process.  The  simulation 
produces  a  number  of  different  log  files  and  an  automated  process  for  aggregating  the  log  files 
into  a  single  log  file  was  created  to  facilitate  the  analysis  process.  Automated  scripts  for 
calculating  many  of  the  descriptive  statistics  for  data  analysis  (such  as  average  reaction  times,  hit 
rates  and  false  alarm  rates)  are  currently  being  created. 

In  summary,  after  pilot  testing  the  baseline  condition  of  the  study,  a  number  of  issues  were 
identified  and  resolved.  Currently,  the  baseline  condition  of  the  GCS  simulator  is  stable  and 
ready  for  the  running  of  participants.  Automated  methods  for  aggregating  and  analyzing  the  data 
are  being  created  and  will  facilitate  the  examination  of  data  once  data  is  collected.  The  remaining 
challenges  over  the  coming  months  will  be  the  efficient  scheduling  of  participants,  and  reviewing 
the  log  files  for  any  errors  or  bugs. 


3  Multimodal  GCS  Interface 


In  the  last  contract  report  (Giang  et  al.,  2011),  a  number  of  ideas  were  proposed  for  the  auditory 
and  tactile  interfaces  for  the  multimodal  GCS  interface.  This  section  of  the  report  will  describe 
the  finalized  auditory  and  tactile  displays  and  their  implementation  in  the  multimodal  condition  of 
the  experiment. 

3.1  Auditory  Sonification  of  Engine  RPM 

An  auditory  sonification  was  used  to  represent  the  current  state  of  engine  RPM.  Two  types  of 
events  are  important  to  monitor  with  regard  to  engine  RPM,  low  engine  RPM  and  high  engine 
RPM.  When  the  UAV’s  engine  RPM  becomes  too  low,  it  is  possible  that  the  engine  will  no 
longer  provide  enough  thrust  and  the  UAV  will  stall;  thus  troubleshooting  a  low  engine  RPM 
event  should  be  a  high  priority  for  the  GCS  operator.  In  contrast,  when  the  UAV’s  engine  RPM 
becomes  too  high,  it  may  cause  the  engine  to  burnout,  which  is  also  an  important  event  to  monitor 
for,  but  not  as  high  of  a  priority  as  a  low  engine  RPM  event.  Normally,  during  steady  flight,  the 
UAV’s  engine  RPM  should  remain  at  a  steady  value.  Deviation  from  this  normal  value  may 
indicate  potential  problems  with  the  UAV. 

An  auditory  sonification  was  created  to  display  the  engine  RPM  value  to  the  operator  using  the 
auditory  modality.  A  sample  clip  of  an  UAV  engine  was  analyzed  using  a  fast  Fourier  transform 
to  determine  its  constituent  frequencies  and  their  amplitudes.  A  sonification  was  created  to 
reproduce  the  frequency  spectrum  of  the  UAV  engine,  but  with  the  ability  to  adjust  the 
fundamental  frequency  and  its  harmonics  (pitch).  The  pitch  of  the  engine  RPM  sonification  was 
mapped  to  the  current  status  of  the  engine  RPM.  As  the  engine  RPM  increases,  the  fundamental 
frequency  and  the  harmonics  of  the  sonification  also  increase  to  reflect  the  status  of  the  engine  in 
real  time.  Similarly,  as  the  engine  RPM  falls,  the  pitch  of  the  sonification  also  decreases.  The 
base  fundamental  frequency,  used  for  normal  engine  RPM  conditions,  is  107.2  Hz,  and  this 
represented  an  engine  RPM  of  2350.  Currently,  the  transformation  between  engine  RPM  value 
and  frequency  is: 

RPM 


In  the  baseline  condition  of  the  experiment,  yellow  and  red  visual  warnings  appear  when  the 
engine  RPM  rises  or  falls  past  certain  thresholds.  The  cautionary  yellow  warning  appears  when 
the  engine  RPM  value  increases  or  decreases  to  a  level  that  the  operator  should  be  concerned 
about.  The  emergency  red  warning  appears  when  the  engine  RPM  increases  or  decreases  to  a 
level  that  requires  immediate  attention. 

We  specified  the  thresholds  for  the  low  engine  RPM  warnings  to  be  2150  (for  a  yellow  warning) 
and  2050  (for  a  red  warning).  For  the  high  engine  RPM,  the  thresholds  are  2550  (for  a  yellow 
warning)  and  2750  (for  a  red  warning).  For  experimental  purposes,  we  also  constrained  the 
engine  RPM  to  have  low  and  high  limits  (1900  for  the  low  engine  RPM  and  2825  for  the  high 
engine  RPM  warnings).  These  limits  were  put  in  place  such  that  a  low  engine  RPM  event  would 
be  in  the  red  warning  region  for  a  longer  duration  than  the  yellow  warning,  while  the  high  engine 
RPM  event  would  be  in  the  yellow  region  for  longer  than  the  red;  this  reinforces  the  urgency  of 
the  low  engine  RPM  events.  These  mappings  are  summarized  in  Table  1. 


Table  1.  RPM  level  mapping  to  visual  and  auditor) >  indications 


RPM  Level 

RPM  Reading 

Visual  Warning 

Auditory  Alarm 

Low  Low 

1900  to  2050 

Red  (Low  RPM) 

Most  urgent  alarm 

FTigh  Low 

2050  to  2150 

Yellow  (Low  RPM) 

No  alarm 

Normal 

2150  to  2550 

None 

No  alarm 

Low  FFigh 

2550  to  2750 

Yellow  (Fligh 

RPM) 

No  alarm 

Fligh  Fligh 

2750  to  2825 

Red  (High  RPM) 

Second  most  urgent  alarm 

Another  important  aspect  of  the  RPM  sonification  was  the  conveying  of  auditory  urgency 
(Edworthy,  Loxley,  &  Dennis,  1991).  For  this  experiment,  it  was  important  that  the  sonification 
conveyed  that  low  engine  RPM  events  are  more  urgent  relative  compared  to  high  engine  RPM 
events.  To  incorporate  the  communication  of  urgency  into  the  sonification,  we  added  warning 
sounds  over  the  original  engine  RPM  sonification  (which  we  refer  to  as  overlay  alarms). 

Auditory  alarms  were  created  by  selecting  two  auditory  alarms  from  Experiment  8  of  Edworthy  et 
al.  (1991).  In  this  experiment,  Edworthy  et  al.  examined  the  perceived  urgency  of  a  number  of 
different  auditory  characteristics  in  different  alarms.  Two  alarms  of  different  urgencies  (the  most 
urgent  alarm  and  the  6th  most  urgent  alarm)  were  selected  as  overlay  alarms.  The  most  urgent 
alarm  was  used  for  low  engine  RPM  events.  When  the  engine  RPM  dropped  into  the  red  warning 
region  (below  2050  RPM),  the  sonification  produced  both  the  engine  RPM  and  the  most  urgent 
auditory  alarm.  The  auditory  alarm  continued  until  the  engine  RPM  decreased  below  2050  RPM. 
Similarly,  the  second  auditory  alarm  (which  was  less  urgent)  was  heard  with  the  sonification 
when  the  engine  RPM  went  above  2750  RPM,  which  was  the  threshold  for  the  high  engine  RPM 
red  warning.  The  signal  to  noise  ratios  for  the  auditory  alarms  and  the  base  sonification  have  not 
yet  been  finalized. 

In  summary,  in  addition  to  the  visual  alerts,  operators  will  be  alerted  to  changes  in  engine  RPM 
using  two  auditory  methods.  First,  the  pitch  of  the  sonification  will  indicate  the  current  status  of 
the  engine  RPM.  Fligher  pitch  represents  higher  engine  RPM  and  lower  pitch  represents  lower 
engine  RPM.  Changes  in  pitch  should  provide  the  operator  with  a  cue  that  the  engine  RPM  is 
deviating  from  normal.  Second,  when  the  engine  RPM  reaches  the  thresholds  for  red  warnings, 
overlay  alarms  will  be  played  to  indicate  to  the  operator  that  the  engine  RPM  has  reached  a  level 
that  requires  immediate  attention.  In  a  previous  study,  Edworthy  showed  that  there  is  a 
relationship  between  warning  sound  parameters  and  perceived  urgency.  A  more  urgent  sound  will 
be  used  for  low  engine  RPM  events  because  of  the  higher  priority  of  such  events  when  compared 
to  high  engine  RPM  events  (Edworthy  et  al.,  1991). 

3.2  Tactile  Attitude  Upset  Display 

A  tactile  display  was  chosen  to  represent  changes  in  the  UAV’s  attitude.  During  flight,  a  UAV 
may  encounter  turbulence  or  wind  shear  that  will  change  the  attitude  of  the  vehicle.  Attitude  upset 
refers  to  any  changes  to  the  aircrafts  pitch,  roll,  and  yaw  from  an  intended  position.  Attitude 
upsets  are  important  to  communicate  to  the  UAV  operator  because  they  may  require  immediate 
action  from  the  operator.  For  example,  during  a  landing,  a  severe  attitude  upset  may  require  that 
an  operator  abort  the  landing  of  the  UAV,  while  even  light  wind  shear  and  turbulence  may 
warrant  operator  concern.  Flowever,  unlike  a  human  piloted  aircraft,  the  UAV  operator  does  not 
have  any  tactile  or  vestibular  feedback  associated  with  the  UAV’s  attitude.  Thus,  a  tactile  vest 
will  be  used  to  communicate  real-time  attitude  upset  information  to  the  operator.  The  display  will 
present  an  aggregated  value  representing  the  magnitude  of  the  attitude  upset. 


To  design  the  tactile  display,  we  first  examined  the  effect  of  varying  levels  of  wind  shear  and 
turbulence  on  the  attitude  of  the  UAV.  Logs  from  the  GCS  simulator  provided  roll,  pitch  and  yaw 
information  from  the  UAV  as  it  encountered  wind  shear  and  turbulence.  A  set  of  graphs  were 
generated  from  these  logs  to  demonstrate  the  roll,  pitch  and  yaw  information  in  different  levels  of 
wind  shear  and  turbulence.  Four  magnitudes  of  the  events,  which  are  used  in  experiment 
scenarios  were  examined  and  indicated  in  Figure  1 . 
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Figure  1.  Graph  of  Varying  Levels  of  Wind  shear  and  Turbulence 


From  these  logs,  it  was  decided  that  the  magnitude  of  the  aggregate  attitude  upset  value  should  be 
based  on  the  change  in  the  rate  of  roll  and  pitch,  and  exclude  measures  of  yaw.  Y aw  was  not  used 
because  changes  in  yaw  were  much  more  likely  to  be  associated  with  the  pre-planned  path 
changes  during  the  recovery  of  the  UAV  than  attitude  upset  from  wind  shear  or  turbulence.  The 
tactile  display  will  use  the  following  calculation  to  determine  the  magnitude  of  the  attitude 
change: 


attitude  upset  = 


Q  Roll  rate  I  +  \Pitchrate\ ) 
2 


The  attitude  upset  information  will  be  displayed  using  an  array  of  tactors,  which  are  small  vibro- 
tactile  motors  which  provide  tactile  stimulation  to  a  single  location  on  the  skin,  located  on  the 
participant’s  back.  A  sample  configuration  can  be  seen  in  Figure  2.  The  display  shows  four  levels 
of  attitude  upset:  a  null  zone  where  no  tactors  are  activated  and  three  levels  of  activation. 


Figure  2.  Arrangement  of  factors. 


The  attitude  upset  thresholds  for  each  level  are  shown  in  Table  2. 


Table  2.  Attitude  Upset  to  Level  Mapping 


Attitude  Upset 

Level 

0-1.5 

Null  Zone  (No  Activation) 

1.5-3 

Level  1 

3-16 

Level  2 

16+ 

Level  3 

The  configuration  of  the  tactors  and  the  tactile  patterns  used  are  still  to  be  determined  based  on 
the  results  of  a  separate  experiment  that  will  be  run  in  the  coming  months.  Two  tactile  patterns 
and  two  layouts  are  being  evaluated  for  use,  and  only  one  pattern  and  one  layout  will  be  selected 
for  use  in  the  multimodal  GCS  experiment.  The  first  tactile  pattern  is  referred  to  as  the 
“sequence”  activation  pattern  (Giang  et  al,  2011).  In  this  method  of  activation,  the  vertical 
distance  of  the  activated  tactor  from  the  body’s  transverse  plane  presents  the  magnitude  of  the 
deviation.  For  example  when  the  magnitude  of  the  UAV’s  deviation  changes  from  Level  1  to 
Level  2,  the  Level  1  tactor  is  deactivated  and  the  Level  2  tactor  is  activated.  The  second  tactile 
pattern  is  what  we  have  called  the  “equalizer”  activation  pattern  (Giang  et  al,  2011).  With  this 
type  of  activation  the  number  of  activated  tactors  represents  the  magnitude  of  the  deviation.  With 
the  equalizer  method,  the  tactor  located  on  the  lower  level  is  not  deactivated  when  the  track 
deviation  increases  from  one  level  to  another.  The  two  layouts  are  either  a  single  3  tactor  column 
or  two  columns  of  3  tactors. 

In  summary,  the  tactile  attitude  upset  display  is  used  to  show  the  UAV  operator  when  the  UAV  is 
experiencing  an  attitude  upset  due  to  wind  shear  or  turbulence.  An  aggregate  value  for  attitude 
upset  is  calculated  using  an  average  of  the  absolute  value  of  rate  of  roll  change  and  rate  of  pitch 
change.  This  aggregate  attitude  upset  value  will  then  be  measured  in  real  time  during  a  UAV 
flight.  If  this  value  exceeds  our  pre-determined  attitude  thresholds,  the  tactors  on  a  tactile  vest 
worn  by  the  participant  will  activate,  displaying  the  information  to  the  participant.  Further  testing 
of  the  tactile  attitude  upset  display  is  still  required  to  determine  whether  the  level  thresholds  are 
appropriate  and  the  type  of  pattern  and  layout  to  be  used. 


3.3  Tactile  Experiment 


The  effectiveness  of  various  tactile  displays  will  be  evaluated  to  examine  how  well  they  are  able 
to  communicate  levels  of  tactile  upset.  This  additional  experiment  was  planned  and  designed  to 
compare  the  sequential  and  equalizer  attitude  upset  presentation. 

Four  display  types  will  be  tested  which  vary  along  two  dimensions:  the  number  of  columns  (1  vs. 
2)  and  the  activation  type  (equalizer  vs.  sequential).  All  four  designs  display  the  magnitude  of  the 
attitude  deviation  using  a  “bar-graph”  like  tactile  display  along  the  upper  back  of  the  participant. 
The  tactile  display  is  comprised  of  tactors  placed  within  a  vest  worn  by  the  participant.  The 
tactors  will  be  arranged  into  columns  of  four  tactors,  with  either  1  or  2  columns  present  on  the 
back  depending  on  the  type  of  display  used.  Four  tactors  are  used  for  each  column  to  ensure  that 
each  tactor  is  easily  localizable.  The  spatial  acuity  of  the  back  is  approximately  3  cm,  and  each 
tactor  will  be  placed  3  cm  away  from  either  other  (edge-to-edge). 

3.3.1  Design 

Two  tasks  will  be  used,  one  to  examine  the  discriminability  of  the  displays  and  one  to  gauge  the 
perceived  urgency.  After  completion  of  both  tasks,  a  questionnaire  about  the  annoyance  level  of 
each  of  the  designs  will  be  provided  to  the  participants. 

3.3.2  Discriminability  Task 

The  discriminability  task  will  examine  how  well  participants  are  able  to  detect  different  levels  of 
attitude  deviation.  The  discriminability  task  will  be  divided  into  two  parts.  The  first  part  examines 
how  well  participants  are  able  to  detect  and  localize  the  activation  of  a  single  level.  This  will  also 
serve  as  a  baseline  for  the  second  part  which  will  focus  on  how  well  participants  are  able  to 
discriminate  between  different  levels  after  a  tactile  pattern  is  played. 

Participants  will  first  be  familiarized  with  the  four  tactile  displays.  Participants  will  experience 
the  two  different  activation  types  (sequential  vs.  equalizer)  and  the  two  layout  types  (one  vs.  two 
columns).  Participants  will  then  experience  examples  of  each  of  the  four  levels  for  each  of  the 
displays. 

After  this  familiarization  phase,  participants  will  begin  the  single  level  activation  section.  In  this 
part,  participants  will  be  presented  with  a  single  level  of  activation  for  one  of  the  displays. 
Participants  will  be  asked  to  respond  the  level  of  activation  presented  using  a  keyboard,  and  will 
indicate  the  level  of  activation  by  pressing  the  appropriate  number  key  (1,  2,  3,  or  4).  Participants 
will  be  asked  to  respond  as  quickly  as  possible.  Response  times  and  accuracy  will  be  recorded. 
Each  condition  will  be  replicated  10  times  and  the  order  of  presentation  will  be  randomized.  This 
section  should  take  approximately  1 1  minutes  to  complete  (160  trials  x  4  sec  per  trial). 

The  second  part  will  examine  participant’s  ability  to  discriminate  between  different  levels  of 
activation  after  they  are  presented  a  tactile  pattern.  This  part  will  test  whether  a  tactile  pattern 
improves  or  hinders  discrimination,  and  also  test  the  effects  of  habituation  and  adaptation.  Two 
types  of  tactile  patterns  will  be  used  and  are  shown  in  Figure  3. 


0123  01234 

Activation  Time  Activation  Time 


Figure  3.  Left  -  "Up"  activation  sequence  for  the  "sequential”  display.  Right  -  "Up-Down"  pattern 
of  activation  for  the  "equalizer"  display. 

In  the  first  pattern,  tactors  activate  one  at  a  time  to  the  target  level  in  sequence,  with  an  inter- 
tactor  duration  set  between  each  level  of  activation;  this  type  of  pattern  is  named  the  “up  pattern”. 
The  second  pattern  is  very  similar  to  the  first,  but  instead  of  stopping  at  the  target  level,  the 
pattern  will  first  go  to  the  next  level  before  returning  to  the  target  level;  this  type  of  pattern  is 
named  the  “up-down”  pattern.  The  participant’s  task  is  to  indicate  the  level  of  activation  after  the 
pattern  as  stabilized  and  they  will  be  asked  to  do  this  as  quickly  and  as  accurately  as  possible. 
Participants  will  indicate  the  final  level  of  activation  using  a  keyboard  by  pressing  the  appropriate 
number  key  (0,  1,  2,  3,  or  4).  The  independent  variables  are  the  type  of  pattern,  the  final  level  of 
activation,  the  inter-tactor  duration  (3  levels  which  still  need  to  be  determined),  and  the  display 
type  (4  types).  The  dependent  variables  will  be  response  time  and  accuracy.  Each  condition  will 
be  replicated  5  times  and  the  order  of  presentation  will  be  randomized.  This  task  is  estimated  to 
take  approximately  80  minutes 

3.3.3  Urgency  Task 

The  goal  of  this  task  will  be  to  determine  the  perceived  urgency  of  the  four  display  types,  the 
different  locations  of  tactors  (which  are  represented  by  different  levels  of  activation),  and 
different  inter-tactor  durations.  The  urgency  task  will  also  be  divided  into  two  parts:  the  first  with 
single  tactor  activations,  and  the  second  with  tactile  patterns. 

In  the  first  part,  participants  will  be  presented  with  a  single  level  of  activation  of  one  of  the 
displays  for  5  seconds.  Participants  will  then  be  asked  to  report  the  perceived  urgency  of  the 
stimuli  on  a  scale  between  1  and  100  using  a  scale  adopted  from  Arrabito,  Mondor,  and  Kent 
(2004)  and  shown  in  Figure  4.  Each  condition  will  be  repeated  5  times  and  the  order  of 
presentation  will  be  randomized.  This  part  is  estimated  to  take  approximately  15  minutes  to 
complete  (80  trials  x  ~10  sec  per  trial). 


Figure  4.  Urgency  Reporting  Screen. 


In  the  second  part,  participants  will  be  presented  tactile  patterns  and  will  be  asked  to  report  the 
perceived  urgency.  Because  the  equalizer  design  requires  a  starting  point,  two  types  of  equalizer 
designs  will  be  tested.  For  the  buttom-up  design,  the  activation  of  Level  1  starts  at  the  bottom 
tactor,  the  activation  of  Level  2  occurs  at  the  middle  tactor  and  finally  the  activation  of  Level  4 
occurs  at  the  top  tactor,  and  vice  versus,  the  top-down  design  starts  at  the  top  tactor.  The  goal  of 
these  two  designs  is  to  examine  if  perceived  urgency  can  be  affected  by  the  starting/ending  point 
of  the  tactor  activation. 

The  independent  variables  will  be  the  final  level  of  activation,  the  inter-tactor  duration  (3  levels 
which  still  need  to  be  determined),  and  the  display  type  (6  types  (one  column  +  two  column, 
bottom-up  equalizer,  top-down  equalizer,  sequential).  The  dependent  variable  is  the  ratings  of 
perceived  urgency.  Each  condition  will  be  replicated  5  times  and  the  order  of  presentation  will  be 
randomized.  This  section  should  take  approximately  (4(level)*3(ITD)*6(display  type)  =  72 
conditions,  360  trials  *  ~  lOsec  per  trial  =  60  minutes). 


4  Implementation  of  Multimodal  Displays  in  the  GCS 
Simulation 


Two  multimodal  displays  were  integrated  into  the  code  of  the  GCS  simulation.  Stub  code  for 
interfacing  with  the  Tucker  Davis  Technologies  (TDT)  System  3  and  the  Engineering  Acoustics 
Inc.  (EAI)  Tactor  Unit  for  the  multimodal  displays  was  added  to  the  GCS  simulation  code 
(Tucker  Davis,  2011). 


4.1  Auditory  Sonification 

The  TDT  was  used  as  the  sound  generator  integrated  into  the  GCS  Simulation.  The  service 
supplier  also  provided  a  visualized  auditory  circuit  development  environment  named  the  System 
3.  The  team  successfully  built  an  auditory  circuit  (shown  in  Figure  5)  under  the  System  3 
development  environment  to  create  the  sonification  as  well  as  overlay  alarms.  Overlay  alarms 
were  developed  using  .wav  files  varying  in  high  urgency  sounds  provided  to  us  by  Dr.  Judy 
Edworthy.  These  sounds  were  used  in  her  research  (Edworthy  et  al.,  1991).  We  established  six 
links  in  the  circuit.  The  links  (shown  in  Table  3)  are  used  to  turn  alarms  on  and  off  and  to  adjust 
the  TDT  System  3  volumes  in  the  GCS  code.  The  volumes  include  control  over  the  signal  to 
noise  ratio  of  the  overlay  sounds  and  the  engine  RPM  sonification  volume. 


Figure  5.  Tucker-Davies  Technologies  A  uditory  Circuit 


Table  3.  Auditory’  Links 


Links 

Level 

LW 

Turn  the  Low  Engine  RPM  alarm  on  or  off 

HW 

Turn  the  High  Engine  RPM  alarm  on  or  off 

LWVol 

Increase  or  decrease  Low  Engine  RPM  volume 

HWVol 

Increase  or  decrease  High  Engine  RPM  volume 

Freq 

Set  the  fundamental  frequency  of  the  engine  RPM 
sonification 

FreqVol 

Increase  or  decrease  engine  RPM  sonification 

4.2  Tactile  Information 


The  EAI  Tactor  Unit  came  equipped  with  a  visualized  development  environment,  the  TAction 
Writer.  Tactile  patterns  were  incorporated  and  played  in  the  GCS  simulation  by  function  calls  in 
the  GCS  code.  An  attitude  monitor  was  embedded  into  the  code.  The  GCS  simulation  monitored 
the  magnitude  of  attitude  upset,  and  played  specific  tactile  patterns  on  the  Tactor  Unit.  The  tactors 
presented  the  tactile  signals  and  the  participant  received  the  stimuli  through  the  vest. 


5  Summary 


In  conclusion,  the  project  made  significant  progress  from  March  of  201 1  to  December  of  201 1.  A 
number  of  stability  issues  with  the  experimental  platform  have  been  resolved  that  are  important 
for  both  the  baseline  and  multimodal  conditions.  As  of  December  2011,  the  baseline  condition  is 
being  run.  In  addition,  the  design  of  the  stimuli  for  the  multimodal  GCS  condition  was  finalized 
and  the  preliminary  implementation  of  multimodal  output  from  the  simulator  was  working  well. 
The  auditory  circuit  changes  the  pitch  of  the  engine  RPM  sonification  correspondingly  with  the 
values  in  the  engine  RPM  indicator.  The  tactile  display  responds  to  attitude  upsets  at  pre¬ 
determined  upset  thresholds  as  designed.  In  early  2012,  the  goals  of  the  project  were  to:  1) 
complete  running  of  the  baseline  GCS  condition  for  both  naive  participants  and  participants  with 
flying  experience;  2)  finalize  the  designs  of  the  multimodal  displays,  including  running  of  the 
tactile  display  experiment;  3)  begin  piloting  of  the  multimodal  GCS  condition.  The  team’s 
progress  up  to  this  point  has  made  it  possible  to  accomplish  these  goals  in  the  near  future. 
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List  of  symbols/abbreviations/acronyms/initialisms 


CF 

Canadian  Forces 

DRDC 

Defence  Research  &  Development  Canada 

GCS 

Ground  Control  Station 

JUSTAS 

Joint  Unmanned  Aerial  Vehicle  Surveillance  Target  Acquisition  System 

RPM 

Revolutions  per  minute 

RPvdsEX 

Real-Time  Processor  Visual  Design  Studio 

R&D 

Research  &  Development 

UAV 

Uninhabited  Aerial  Vehicle 

